Back to projects

Case study, retrieval augmented generation

ragx: a docs assistant that cites its sources and says when it does not know

Ask a question about the FastAPI or Pydantic documentation. Every answer links to the pages it came from, off topic questions are refused, and every claim on this page is a measured number with its limits next to it.

Open the live demo View the code

The demo runs on a free server that sleeps when idle, so the first load can take about a minute. Choose Continue as guest to try it without an account.

Four problems, and what happened when I measured the fix

Each card names the problem, what I built, and the number I got. The limits are in the section further down.

  • The problem
    Chatbots answer with confidence even when the documents do not cover the question.
    What I built
    Every answer must cite numbered sources. When nothing relevant is found, it says it does not know and shows the nearest pages it searched.
    Result
    • Refused 100% of the 8 off topic questions and 83% of the 6 in domain questions whose answer is not in the indexed pages.
    • 96% of answers carried a citation, and 0.97 of the citations supported the sentence they sat on (judge scored).
  • The problem
    Keyword search and embedding search each miss pages the other one finds.
    What I built
    Hybrid retrieval: BM25 keyword search and dense embeddings, merged by reciprocal rank fusion. Hits expand to the whole documentation section, so the model sees full context.
    Result
    • On held out questions, nDCG@10 is 0.84 for hybrid against 0.81 for keyword and 0.80 for embeddings.
    • hit@5 is 97% against 89% for either method alone.
  • The problem
    Free model tiers rate limit, change and fail, so one provider is a single point of failure.
    What I built
    A router tries Groq, Cerebras and Gemini in order, with fallback, a circuit breaker and quotas. A paid key exists only as a capped last resort and is off by default. An exact plus semantic cache answers repeats without a model call.
    Result
    • Paid spend in the end to end eval run: $0. The semantic cache served 61% of paraphrased questions with 0% wrong hits at the chosen threshold.
    • The honest cost: on free tiers the median answer took 2.6 s but the slowest 5% took 63.8 s when providers rate limited, and 73% of calls used a fallback in that run.
  • The problem
    Assistants follow instructions hidden in questions and documents, and leak personal data.
    What I built
    Input checks block injection and redact emails, keys and phone numbers before any model sees them. Retrieved chunks that carry instructions are dropped, and outputs are checked. Memory needs consent and rejects secrets.
    Result
    • Blocked 92% of 24 attack prompts, with 0% false refusals on 17 benign look alikes.
    • 0 of 7 personal data cases leaked into a model input, response, log, memory or cache.

See it working

Screenshots from a local run of the same build that is deployed. The questions are real; the answers come from free tier models.

The chat screen answering a question about before and after validators in Pydantic, with numbered citation chips and a list of sources
A cited answer. Each bracketed number opens the exact documentation section the sentence came from.
The chat screen refusing an off topic question about PyTorch and listing the nearest documentation pages it searched
An off topic question is refused, and the assistant shows the nearest pages it searched.
A dialog showing the source passage from the Pydantic validators page with a link to the original page
Opening a citation shows the source passage and links to the original page.
The chat screen on a phone, showing a code block, citation chips and a bottom tab bar
On a phone the navigation moves to a bottom bar and code blocks scroll sideways.
The admin Evals tab, a table of metrics with the result file and date for each one
The admin Evals tab reads the committed result files, so every number on screen names the run it came from.

How a question is answered

Six steps. Each one exists because a measurement or a failure showed it was needed.

  1. Step 1

    Check the question

    Length limit, prompt injection check, and redaction of emails, keys and phone numbers.

  2. Step 2

    Look in the cache

    An exact match or a very similar earlier question returns at once, with no model call.

  3. Step 3

    Search the docs

    BM25 and embeddings run side by side and are merged by rank. Results expand to whole sections.

  4. Step 4

    Ask a model

    A router picks a free provider and falls back if it is rate limited or down.

  5. Step 5

    Check the answer

    Citations must point at real sources, and personal data is removed from the reply.

  6. Step 6

    Measure everything

    Each query is traced, counted in Prometheus, and a sample is scored by a judge model.

Evals: what I measured

One change at a time, on a fixed set of 128 golden questions (90 to tune on, 38 held out and never used to choose settings). Results that got worse are kept and shown. Every chart names its source file in the repository.

Retrieval: hybrid search wins, and the reranker made it worse

nDCG@10 per retrieval mode. Higher is better. Hybrid also scored hit@5 of 97% on the held out questions. A cross encoder reranker scored lower and took about 1.9 s per search instead of 16 ms, so I left it out.

  • Dev split (90 questions)
  • Held out (38 questions)

  • Keyword (BM25)

    median search 14 ms

    0.82
    0.81

  • Dense embeddings

    median search 10 ms

    0.83
    0.80

  • Hybrid (shipped)

    median search 16 ms

    0.88
    0.84

  • Hybrid plus reranker

    median search 1.9 s

    0.80
    0.79

  • Hybrid plus MMR

    median search 12 ms

    0.84
    0.82

Source: eval/results/01-retrieval.json

Self RAG against plain RAG: more calls, no better answers

Self RAG adds retrieval grading and answer checking. On 43 questions it used 3.9 model calls and 2,548 tokens per question against 1 call and 1,188 tokens, and was not more faithful, so plain RAG is the default. Differences of a few points on this many questions are within noise.

  • Plain RAG (shipped)
  • Self RAG

  • Faithfulness

    claims supported by the sources

    0.93
    0.91

  • Citation accuracy

    citations that support their sentence

    0.97
    0.96

  • Answers with a citation

    0.96
    1.00

  • Refusal correct

    answers when it should, refuses when it should

    0.88
    0.93

Source: eval/results/02-generation-v3-bracketfix.json

Safety rules, before and after one round of fixes

The second round changed rules after seeing which cases failed, and the same cases are the test set, so the after numbers are partly tuned on their own test data. Two social engineering attacks are still missed.

  • Round 1
  • Round 2

  • Attack prompts blocked

    24 attacks, higher is better

    83.3%
    91.7%

  • Benign questions wrongly refused

    17 look alikes, lower is better

    11.8%
    0.0%

  • Personal data found exactly

    12 cases, higher is better

    83.3%
    100.0%

Source: eval/results/04-safety-v1.json and eval/results/04-safety.json

Conversation memory helps follow up questions

Rewriting a follow up like “and how do I validate it?” using the earlier turns. Retrieval only, 12 cases written by Claude, so treat it as an indication.

  • Without memory
  • With memory

  • hit@5

    0.58
    0.83

  • Mean reciprocal rank

    0.37
    0.71

Source: eval/results/04-memory.json

Voice and image input

Tested on 18 synthetic voice clips and 18 rendered images, so these numbers are optimistic. A glossary prompt lowered the word error rate. No hidden instruction in an image reached the pipeline.

  • Word error rate, plain

    lower is better

    6.1%

  • Word error rate, with glossary

    lower is better

    4.6%

  • Key terms read from images

    0.91

  • Right page in the top five from an image

    0.94

Source: eval/results/05-voice-glossary.json, eval/results/05-voice-plain.json and eval/results/05-image.json

Approximate search indexes at 100,000 vectors

A study of scale, not of this app: the real corpus has only 907 chunks, where exact search is already fast. At 100,000 synthetic vectors, HNSW and IVF settings that reach full recall run roughly 15 to 60 times faster than exact search (up and to the left is better).

  • Exact
  • HNSW
  • IVF
0.920.950.981.000.1 ms1 ms10 msMedian search time, log scaleRecall at 10Flat (exact): recall 1.000, 9.24 msHNSW M16 ef16: recall 0.951, 0.082 msHNSW M16 ef64: recall 1.000, 0.151 msHNSW M16 ef256: recall 1.000, 0.527 msHNSW M32 ef16: recall 0.967, 0.096 msHNSW M32 ef64: recall 1.000, 0.193 msHNSW M32 ef256: recall 1.000, 0.618 msIVF nlist256 nprobe1: recall 0.997, 0.121 msIVF nlist256 nprobe8: recall 1.000, 0.588 msIVF nlist256 nprobe32: recall 1.000, 2.213 msIVF nlist1024 nprobe1: recall 0.932, 0.123 msIVF nlist1024 nprobe8: recall 1.000, 0.24 msIVF nlist1024 nprobe32: recall 1.000, 0.655 ms

Source: eval/results/01-ann.json

Context precision, judged by a model

0.81

Rank weighted share of the sources sent to the answer model that are useful for the question, over all 128 golden questions. Dev 0.81, held out 0.81. 50% of sources were judged useful and 95% of questions had at least one.

The judge agrees with the page label only 74% of the time, and I have not calibrated it by hand for this task, so read this as a rough signal. Context recall is not measured because the golden set has no reference answers.

Source: eval/results/01-context-precision.json

Personal data leak test

0 of 7

Cases with an email, key, token or phone number went through the real app with a stub model built to behave as badly as possible: it echoes everything it receives and tries to save the user's words as a memory. I then searched the model input, the response, the trace log, the audit log, memories, history and the cache. Nothing leaked, and none leaked from text read out of a screenshot.

A control run with the guards switched off leaked 7 of 7 cases through the same harness, so the zero is not a blind spot. Limits: seven cases, only patterns the rules can detect (names and addresses are not detected), and a stub, not a real model.

Source: eval/results/04-pii-leak.json

Watching it run

Prometheus scrapes the service and Grafana draws the online evals, so quality is watched after release as well as before.

Grafana panels showing questions per minute by outcome, refusal rate, answer latency and tokens per minute
Questions per minute by outcome, refusal rate, answer latency (median and 95th percentile) and tokens per minute.
Grafana panels showing sampled online faithfulness, the number of judged samples and paid spend of zero dollars
A sample of live answers is scored by a judge model after the reply is sent. Here every answer was scored, and paid spend stays at zero dollars.

These panels come from a local run with a short burst of test questions, not production traffic. In the deployed service the sample rate defaults to 5% of answers. The alert thresholds are placeholders until a few weeks of real traffic can tune them.

What did not work, and what is not proven

The failures are part of the result. A portfolio that only shows wins is hard to trust.

  • The reranker lowered quality and added seconds

    It scored below plain hybrid search on nDCG@10 and took 1.9 s a search, so it is not in the product.

  • Self RAG was not worth its cost

    About 4 times the model calls for no gain in faithfulness on this set.

  • Free tiers are slow when they push back

    Median 2.6 s, but 63.8 s at the 95th percentile when providers rate limited. A paid key would fix that and is deliberately off.

  • The labels are unreviewed

    Beyond the 30 baseline questions, the golden questions and all safety labels were written by an AI assistant (Claude) and a human has not reviewed them. The judge model agreed with its calibration labels on 30 of 30 cases, but with unreviewed labels that is weak evidence.

  • Voice and image tests are synthetic

    Generated voices and rendered images are cleaner than real ones. Real phone recordings and real screenshots would score lower.

  • Not measured yet

    Context recall, a personal data leak rate on live answers with a real model, and two completed scheduled eval runs. Names and addresses are not detected as personal data.

Built like a service, not a notebook

  • FastAPI backend with argon2 password hashing, short lived tokens with refresh rotation, role based access, rate limits and an audit log.
  • A web app that installs like an app, with voice and screenshot input, history, memory with consent and an admin area.
  • 139 automated tests. GitHub Actions run them on every push and run the evals on a schedule, and a regression gate compares each run with the last good one.
  • A container image that runs as a non root user, deployed on a free Render service, with secrets only in the host settings.
  • Docs pinned to exact upstream commits, so every eval can be repeated on the same text.

Stack

  • Python
  • FastAPI
  • BM25
  • FAISS
  • bge small embeddings
  • Groq
  • Cerebras
  • Gemini
  • Prometheus
  • Grafana
  • Docker
  • GitHub Actions
  • Render

Numbers on this page were read from the result files in the repository at commit ec534b7, not typed by hand.

Live demo View code