Sage Training — Phase 1 Complete

Log every question where Sage's top KB result was low-confidence or where the user thumbs-downed the answer. Surface these as a weekly "questions Sage can't answer well" report. This tells you exactly which articles to write or improve next, instead of guessing. The conversations table already tracks this data — just need a dashboard view.

Effort: 1 day — new admin endpoint + page. Payoff: ongoing, compounding.

Before a user's question hits UserGuiding search, rewrite acronyms and aliases: BP → BenefitPoint, RFP → Request for Proposal, MR → Market Response, Brite → Brite Benefits. Keyword search fails silently when the user's wording doesn't match the article. A 30-term glossary fixes most misses.

Effort: 2 hours. Payoff: instant improvement on ~15-20% of queries.

UserGuiding returns results by keyword score, which is noisy. Fetch the top 8 articles, have Claude score each one for relevance to the actual question, then send only the top 3 as context. Fewer tokens, sharper answers, less "confidently wrong" responses. Cost: one extra small Claude call per question.

Effort: half a day. Payoff: answer quality jump on ambiguous questions.

4. Conversation memory for follow-ups

Right now every question is treated as standalone. Track the last 2-3 turns so users can ask "what about for admins?" or "and for enterprise?" and Sage keeps context. The conversations table already exists — this is wiring it into the prompt.

Effort: half a day. Payoff: feels dramatically more like talking to a human.

5. Confidence-gated responses

When Sage's retrieved context is weak (low relevance scores, or the KB returned nothing), she should say "I'm not certain — want me to flag this for the team?" instead of improvising. Improvising is where hallucinations come from. Tie this to the HubSpot ticket creation flow that's already partially built.

Effort: 1 day (includes finishing HubSpot integration). Payoff: eliminates the scariest failure mode.

6. Semantic search layer (embeddings)

UserGuiding search is keyword-based — it misses paraphrases. Add an embedding index on top so "invite someone to my workspace" matches "add a team member." Voyage AI or OpenAI embeddings, stored in Neon's pgvector extension. Runs alongside MCP search; results merged.

Effort: 2-3 days. Payoff: catches the 20% of queries keyword search misses entirely.

7. Video timestamp deep-linking

For the 16 video-enhanced articles, when Sage cites the transcript, link directly to the exact Wistia timestamp instead of the article page. Transcripts already include timestamps — just needs to be passed through to the citation.

Effort: 2 hours. Payoff: small but delightful UX win.

8. Automated feedback → training loop

When a user thumbs-downs an answer, auto-capture the question, the context Sage used, and the response — then queue it in an admin review page where you can add a correction to the Training Knowledge store. Turns every bad answer into a permanent fix.

Effort: 1 day. Payoff: Sage gets better every week without manual work.