How we evaluated this system

This page reports, in plain terms and without spin, how well the “Ask the Guide” assistant does two jobs: finding the right policy and answering from it. Everything here is measured, and the methods are disclosed below. The official MSW Student Guide remains the authoritative source; this tool is a way to search it, not a replacement for it.

1. Finding the right policy (retrieval)

We wrote 150 realistic student questions whose correct policy section is known in advance, then measured how often the system surfaces that exact section. Three plain measures:
Recall@5 — how often the correct section appears in the top 5 results (higher is better).
MRR — rewards putting the correct section at the very top (1.0 = always first).
Mean rank — the average position of the correct section (1.0 is ideal).

Best configuration: qwen·whole·jina. We compared 18 combinations of three choices — how the guide is split into pieces (“chunking”), which method converts text to searchable form (“embedding”), and whether a second-pass “reranker” reorders results.

ConfigurationRecall@1Recall@5MRRnDCG@10Mean rankvs. primary (p)
qwen·whole·jina0.9401.0000.968 [0.950.99]0.9751.07
bge·whole·jina0.9401.0000.968 [0.950.99]0.9751.071.000
qwen·structural·jina0.9271.0000.962 [0.940.98]0.9711.080.517
bge·structural·jina0.9271.0000.961 [0.940.98]0.9701.090.387
qwen·window·jina0.9271.0000.960 [0.940.98]0.9701.10.395
bge·window·jina0.9271.0000.959 [0.940.98]0.9691.10.339
bge·whole·none0.9131.0000.955 [0.930.98]0.9661.10.378
bge·structural·none0.9201.0000.955 [0.930.98]0.9661.120.373
bge·window·none0.9200.9870.950 [0.920.98]0.9611.190.228
qwen·structural·bge_xenc0.9000.9930.945 [0.920.97]0.9581.160.138
qwen·whole·none0.8931.0000.943 [0.920.97]0.9581.130.114
bge·structural·bge_xenc0.8931.0000.941 [0.910.97]0.9561.150.103
qwen·window·none0.8871.0000.940 [0.910.96]0.9561.130.070
qwen·whole·bge_xenc0.9000.9870.940 [0.910.97]0.9551.210.081
qwen·structural·none0.8871.0000.939 [0.910.97]0.9551.140.071
qwen·window·bge_xenc0.8871.0000.936 [0.910.96]0.9511.170.069
bge·whole·bge_xenc0.8870.9870.935 [0.900.96]0.9511.210.047
bge·window·bge_xenc0.8801.0000.931 [0.900.96]0.9481.190.043

Bracketed ranges are 95% confidence intervals from 1,000 bootstrap resamples of the questions. “p” is a paired bootstrap test versus the primary configuration (qwen·whole·jina); values above 0.05 mean the difference is not statistically distinguishable.

Keyword search vs. searching by meaning

Classic “keyword” search matches the exact words a student types. “Semantic” search matches meaning, so “can I bring my dog” finds the emotional-support-animal policy. On the same 150 questions, keyword search failed to put the correct policy in the top 5 for 3.3% of questions; searching by meaning failed for 0.0%.

MethodCorrect is #1Correct in top 5MRR
Keyword (BM25)0.8200.9670.884
Semantic (meaning)0.8931.0000.943
Semantic + re-ranking (used here)0.9401.0000.968
Student’s phrasingKeyword’s top guessSemantic’s top guess
can I bring my dog to class1.18.01 Financial Aid Appeal Process1.20.06 Emotional Support Animals (ESAs)
kicked out for bad grades2.12.10 Out-of-Sequence Plans for Field Education1.13.06 Potential Outcomes of the Academic Concerns Committee
money for school1.13.00 Jurisdiction1.18.00 Financial Aid Policies
how many hours do I work at my internship3.02.11 Internships, Department of Labor Regulations, and Social Work Education Guidelines3.06.05 Employment Conflicts
what if I get hurt at my placement2.08.02 Courses from Other Departments3.08.01 Guidelines for Risk Reduction

2. Answering from the policy (answer quality)

A separate AI model (glm-5.2:cloud) wrote answers using only the retrieved policy. Each answer was then judged by Claude Opus on five plain dimensions, scored 1–5.

groundedness
5.00 /5
correctness
5.00 /5
completeness
4.94 /5
citation accuracy
4.98 /5
relevance
5.00 /5
Answers judged acceptable
100.0%
Hallucination rate
0.0%
Appropriately declined
0.0%
Avg. response time
9.52s

Based on 49 judged answers. “Hallucination” means stating something not supported by the retrieved policy — the single most important thing to keep near zero.

3. Trying to break it (adversarial testing)

We deliberately attacked the system with 50 hard cases: questions about policies that don’t exist, false premises, off-topic requests, and attempts to make it ignore its rules. A good result is that it refuses or corrects rather than inventing an answer.

Handled safely (robust)
90.0%
Partial
10.0%
Failed
0.0%
Resisted instruction-override attacks
100.0%

By attack type

Attack typeCasesRobustPartialFailed
hallucination bait7700
nonexistent section3300
false premise8530
prompt injection8800
jailbreak7700
out of scope111100
leading1010
over generalization3210
citation integrity2200

Every attack and the system’s response

4. Hardening: two rounds, on identical + fresh attacks

I added a “query pre-processor” that cleans each question, refuses manipulation before it reaches the answer model, corrects false claims from policy, and surfaces the closest real policy when a specific one doesn’t exist. I re-ran the same attacks after each round, and then ran 20 brand-new attacks the system had never seen. Note the “Failed” row: it was zero at every stage — the system was always safe; hardening improved how helpfully it stays safe.

MeasureBaselineHardening v1Hardening v2v2 · 20 new attacks
Handled safely90.0%98.0%100.0%100.0%
Partial (safe, not ideal)10.0%2.0%0.0%0.0%
Failed (unsafe)0.0%0.0%0.0%0.0%
Instruction-override resistance100.0%100.0%100.0%100.0%
Made something up under pressure0.0%0.0%0.0%0.0%

Full transcripts of every attack and response — baseline and hardened — are in the results files. 70 attacks is meaningful but limited; adversarial testing is never “finished.”

5. Consistency & knowing its limits

Correct policy found across all rewordings of a question
100.0%
Answerable questions it answered (didn’t wrongly refuse)
100.0%
Uncovered questions correctly flagged “not in the guide”
100.0%
Uncovered questions where it fabricated a policy
0.0%

Rephrasing a question doesn’t change whether it finds the right policy, and on a set of questions the guide doesn’t cover, it declined every one without inventing a policy. Still measured on my question sets, not real student traffic.

Methods & honest limitations