markpoint

The problem

A student revising for A-Level Biology wants two things a search box cannot give them: which past papers a question has appeared in before, and why their written answer lost marks. Generic chatbots answer both badly, inventing paper references and marking generously.

How a question finds its papers

Approach

• Parsed 315 CIE Biology past papers into 10,892 questions and 30,176 atomised mark points, keeping accept and reject variants separate, with per-family parsers keyed on subject, paper and syllabus era. Each parse carries a confidence score, and low-confidence output routes to a human review screen rather than into the corpus.
• Built retrieval in two stages rather than one blended score. Stage A recalls over the whole corpus by reciprocal-rank fusion of Postgres full-text ranking with dense pgvector search, fused by rank because the two scores live on unrelated scales. Stage B reranks the top 50 by IDF-weighted Jaccard over mark-point sets, which is the strongest signal available but cannot be indexed for approximate search, so it can only ever be a reranker.
• Assembled every citation from SQL joins. A model may rank and explain, but it may never emit a paper identifier, which removes hallucinated citations by construction instead of mitigating them by prompt.
• Marked one mark point at a time, each verdict carrying the ID of the point it judged, and ran whole-paper marking concurrently rather than batching points into a single prompt. Batching would break the per-point rule, and models are trained to be generous, which is the exact failure the product exists to correct.

Killing a feature that looked like it worked

A lexical matcher linking multiple-choice questions to the structured questions that explain them produced 128 links. Read one at a time they looked fine. Read as a distribution they pointed at just 17 target questions, one of them 51 times, because Postgres ts_rank_cd carries no document-length term and the longest candidate wins on shared vocabulary regardless of subject. Score did not correlate with quality, so no threshold could fix it. The feature was disabled rather than shipped, with the measurement written into the code.

Stack

• FastAPI
• PostgreSQL 17
• pgvector
• React 19
• Docker