Shen.ai: patient education with safety controls

5 min readSimon BudziakBy Simon Budziak

On this page7 sections

Shen.ai turns smartphone cameras into health monitors that measure more than 30 vital signs from a face scan. Patients needed help understanding what their results meant and what to do next.

An assistant in this setting cannot fill gaps with a plausible answer. It must explain approved information in plain language, keep education separate from diagnosis, and route uncertain or clinical questions to a person.

The problem

The patient-facing assistant needed to:

  • explain health scan results in plain language;
  • answer follow-up questions conversationally;
  • retrieve educational content from an approved medical knowledge base;
  • suggest the appropriate next step, such as self-care information or an appointment;
  • route low-confidence and clinical questions to a clinician;
  • retain enough evidence to reconstruct what the system said and why.

The stakes changed the engineering approach. A confident but unsupported explanation is not a small user experience defect. The team needed explicit safety boundaries and a trace for every decision.

Two risks appeared early.

Invisible hallucinations. Weak retrieval could still produce a fluent, confident response.

Unclear escalation. A prompt asking the model to be cautious did not define when the system had to stop and request clinical review.

The system

We built the assistant as a LangGraph state machine. The state carried the scan findings, retrieved medical context, confidence score, review status, and conversation history.

The critical routing decision lived in code rather than in a prompt.

class PatientEducationState(MessagesState):
    test_results: dict
    confidence_score: float
    requires_review: bool
    thread_id: str

def should_route_to_human(state: PatientEducationState) -> str:
    if state["confidence_score"] < 0.7 or state["requires_review"]:
        return "human_review"
    return "education_module"

graph.add_conditional_edges("retrieve_kb", should_route_to_human)

This kept the boundary visible. The model produced language inside the permitted path, but it did not decide whether a clinical escalation could be skipped.

What made it safe enough to ship

Approved knowledge with confidence scoring

The assistant used retrieval-augmented generation over a version-controlled knowledge base of clinical explanations and patient education material. Entries carried source information, and retrieval combined semantic relevance with source quality.

Confidence gating made uncertainty explicit. Low-confidence results did not enter the normal answer path. They triggered a safe response and clinician review.

The knowledge base followed the same discipline as production code: version history, change tracking, and rollback. That made it possible to answer which medical material the assistant had available during a specific conversation.

Explicit routing to a clinician

Uncertainty became a system state. The initial routing pattern used a combined signal from retrieval quality, model confidence, and specific clinical triggers.

Questions about symptoms, medication, or diagnosis followed the human-in-the-loop review path. The goal was not to answer every question. It was to answer educational questions from approved material and stop safely everywhere else.

After threshold tuning, the observed production pattern was:

  • about 15% of conversations entered human review, mostly edge cases or off-topic questions;
  • clinician feedback judged about 80% of those escalations appropriate;
  • lowering the initial threshold from 0.8 to 0.7 reduced unnecessary reviews by about 40%;
  • weekly audit sampling found a very low rate of inappropriate clinical advice.

These figures describe the tuning period and the review process used for this system. They are not a general benchmark for healthcare assistants.

A trace that explains every answer

LangSmith recorded the model calls, retrieved documents, confidence scores, routing decisions, and run metadata for each conversation. Its observability model kept the steps for each operation in one trace.

That trace served two jobs. Engineers could debug why a response escaped its intended boundary. Reviewers could reconstruct the knowledge version, model version, and routing state behind an answer after the fact.

Session checkpointing in PostgreSQL also let patients leave and return without losing the state required for the safety checks.

The failure that changed the grounding rule

During testing, the assistant confidently explained a condition that was not present in the approved knowledge base.

The trace showed what happened. Retrieval had returned weak matches, but the generated answer did not reflect that uncertainty. The model bridged the gap with information from its training data.

The fix moved the boundary out of the prompt and into the workflow:

  1. Retrieval below the safe confidence threshold could not reach the normal answer node.
  2. The assistant returned that it lacked enough information to answer safely.
  3. The conversation entered clinician review.
  4. A set of known-unknown questions joined the evaluation dataset.

In this system, saying “I do not know” was a required outcome, not a model failure.

The audit gap

An early LLM observability audit could recover the final message but not the full basis for it. That was not enough. Reviewers needed the retrieved material, confidence score, model version, routing decision, and timestamp.

The production trace added stable session identifiers, knowledge-base versions, model versions, structured exports, and redaction rules for monitoring data. The result was a record that could explain both the answer and the path that allowed it.

The result

Shen.ai received a patient education assistant that explains scan results from approved material and treats uncertainty as an explicit route. The system keeps education separate from diagnosis, hands clinical questions to people, and records the evidence behind every production decision.

The central lesson is not to ask a healthcare model to be careful. Define what it may answer, make the stop conditions deterministic, and test the cases where the correct answer is a handoff.

Technical references

Last updated: