Building explainable hybrid search with Go, SQLite and Qdrant

Language: English | 简体中文

I added an explicit hybrid search mode to French Learning Hub so a learner can find a Concept through both its wording and its meaning, then inspect how that result was retrieved. The implementation combines existing SQLite keyword results with Qdrant semantic results, exposes their evidence in React, and checks the final results against current records.

My engineering contribution: defining a bounded ranking contract, integrating the two production search paths, making the result evidence visible, and driving acceptance through an independent ranking oracle.

发布日期：2026-10-14

The problem: two rankings with different meanings

Keyword search is useful when the learner remembers a grammar term or a fragment of an example. Semantic search is useful when the learner remembers a meaning expressed in different words. Neither ordering can simply be added to the other: keyword search returns a deterministic tier/state/title order, while semantic search returns vector similarities.

I used reciprocal rank fusion so the combination depends on positions rather than pretending those scores share a scale. The policy is versioned as concept_hybrid_rrf_v1, uses equal weights and a fixed constant of 60:

score(concept) = 1 / (60 + keyword_rank)
               + 1 / (60 + semantic_rank)

A component contributes only when it retrieved that Concept. Duplicate IDs within one component keep their first rank, and ties use Concept ID ascending. Candidate depth is capped at 50 per component; displayed result count is separate and cannot exceed that depth.

Make the contract visible to the user

The workbench has Keyword, Semantic and Hybrid modes. Each hybrid result shows which components found it, their ranks, keyword matched fields, semantic similarity and the fused score. The response also identifies its policy and semantic publication, and reports dropped candidates.

The keyword component is the production prefix-matching search, not the separate BM25 evaluation baseline. The fused score is a ranking sum, not a confidence probability. Keeping those distinctions in the API and interface makes the output explainable during debugging and review.

Current data remains authoritative

Fusion does not create Concept memberships or change learning records. After both searches complete, the service reads Concepts again from SQLite. A result is dropped if it disappeared, changed identity/state or no longer passes the filter. Missing results are not silently replaced.

The semantic side reads one published collection, using its existing bounded retry when publication changes. If the semantic service or Qdrant fails, hybrid reports that failure. Keyword remains independently available through its own mode. This preserves the meaning of the mode the user selected.

Verification beyond calling the same function twice

Integrated acceptance used the real Go server and CLI, real Qdrant, synthetic records and deterministic fake embeddings. A separate oracle computed the expected ranking from the two component responses rather than invoking the production fusion function.

The recorded 126 adjudicated checks covered fusion arithmetic, ties, filtering, result evidence, stale authority, provider failures, publication replacement, restart recovery and cache reuse. Across 23 stable comparison groups, logical SQLite dumps and the publication pointer stayed unchanged. Keyword made zero provider calls; semantic and hybrid each embedded only one query per request.

The fixture contained 15 Concepts. Rebuild and repair reused their cached document vectors without new document embedding calls. These checks establish ranking mechanics and recovery behavior. Comparing relevance on a human-reviewed query set is a separate next step.

What this project demonstrates

This feature connects algorithm design to service behavior: bounded work, deterministic output, current-state checks, a useful API and an inspectable frontend. The important deliverable is a search mode whose decisions and failure states can be explained.

Implementation used AI coding agents under my task definitions and review. The results above are recorded project acceptance evidence.

Implementation · Design contract · Integrated acceptance · About me

Implementation milestone.