Building with AI
A working AI speaking coach, built solo for a course capstone (Aug–Sep 2025). No team, no real users, no business stakes: what it shows is hands-on fluency with prompt iteration, structured outputs, evals and adversarial testing.
It pairs with the AI Coach case: the same speaking-practice gap, once inside real organizational constraints and once here with full technical freedom.
- Role
- Solo builder: PM, researcher, prompt engineer, evaluator
- Timeline
- Aug–Sep 2025 (PRD Aug 12; capstone presented Sep 20)
- Context
- "AI for Product Managers" course (Miqdad Jaffer, ex-OpenAI, Maven)
- Tools
- Lovable, Supabase, OpenAI (GPT-5 → GPT-4.1-mini), Grok for adversarial test data
Why this exercise
The capstone asked for an AI feature built end to end: discovery, a working prototype and a rollout plan. I didn't invent a toy problem; I used a gap from my job. RealLife English users watch a lesson, do flashcards and writing exercises, but the next real chance to speak is a scheduled session with another person. Nothing bridges "I did the exercise" to "I can use this in a real conversation."
The persona and the gap
Target user: people already working in English-speaking environments, already paying for language solutions, who know the rules but aren't confident because they're not using the vocabulary in real scenarios.
Market context: ESL is at risk of commoditization by AI, with Duolingo, Cake and AI conversation partners like Loora moving fast. Human connection stays relevant; the opportunity is AI as an enabler of practice, not a replacement for real conversation.
"Excellent submission. You've clearly done the work to understand your user's journey, mindset, and pain points... Your emotional UX mapping is excellent... Thoughtful, focused, and clearly driven by real learner insight."Miqdad Jaffer, course instructor, on the discovery phase
His critiques: size the market more tightly, clarify the monetization event, and be explicit about how the LLM powers the feature (generation vs. feedback vs. retrieval). Fair notes on the plan, not on the prototype. Honest tradeoff: with the course clock running, I moved into design and development without acting on that feedback first, and said so at the time.
The prototype
Flow: watch lesson → flashcards → fill in the blanks → AI speaking practice (role play with instant feedback). The AI step was new; the rest mirrors RealLife English, so the coach had to fit an existing mental model.
A deliberate constraint: at most 3 AI exchanges per session. Short sessions, lower cognitive load, and practice about retrieval, not open-ended chat.
Prompt design and evals
I started with GPT-5 for everything, then found GPT-4.1-mini enough for the core loop, and cheaper.
The real work was structured output. Early tests leaked the answer inside the exercise: the model generated a fill-in-the-blank and included the answer in the same response. First eval passes conflicted with each other. I moved from ~20 loosely typed fields to a YAML-structured output and iterated prompt and eval criteria together.
- Exercise generation: structure, order, relevance (uses target vocabulary), no invented vocabulary, tone.
- Feedback generation: tone, relevance, focus on comprehension over grammar nitpicking, accepts semantically valid answers, shadowing prompts, allows retry, no hallucination, stays on topic.
Stress-testing with synthetic personas
Rather than wait for real users, I generated four adversarial personas: common (minor mistakes like "on" vs. "in"), realistic (beginner mistakes like "teached"), angry and unsafe (profanity in English and the user's language, generated with Grok) and off-topic (irrelevant answers, like a TV on in the background).
Synthetic testing caught a real edge case: asked to conjugate the irregular verb "to pay," the model returned a blank ("_ed") instead of "paid off." Small, but the kind of failure that erodes trust fast.
What I'd ship next, and a rollout plan
MVP scope: exercise generation, chat, feedback and an audio interface. Left out: per-word score tracking over time, a live conversational model, and personalization by stated goals.
The two-segment logic mirrors what Daily Missions found: trial-to-renewal behavior is a different signal from trial-start conversion.
What I'd do differently
- Evals tuned on synthetic data missed real hallucinations. With real lessons and vocabulary, the model produced errors the eval set never caught.
- Chat-only felt repetitive. The role-play worked, but the format needs more texture, and should move from performing a scenario to telling a real story.
- I never got to cost and viability. No token-level cost model; I flagged the gap instead of skipping it.
- Compare more models and eval methods, and scope phonetics detection earlier: it needs embeddings or fine-tuning, a different approach from the rest.