Semantic Search

Traditional keyword search matches words. If you search for "employee termination process," it looks for documents containing those exact words. Semantic search understands that "how to offboard someone" means the same thing and returns those documents too.

This works because both the query and the documents are converted into embeddings (numerical vectors) by the same model. Similar meanings produce similar vectors. The search system finds documents whose vectors are closest to the query vector, typically measured by cosine similarity or dot product, regardless of the specific words used.

The practical impact is significant. Enterprise knowledge bases are full of documents written by different people using different terminology. A policy might say "involuntary separation" while the search query says "firing someone." Keyword search misses this. Semantic search catches it.

Most production systems use hybrid search, combining semantic and keyword approaches. Semantic search is better at understanding intent. Keyword search is better at finding specific names, codes, product SKUs, or identifiers where the exact token matters. A hybrid approach uses both signals (often combined via Reciprocal Rank Fusion) to rank results, getting the best of each. On enterprise eval sets, hybrid typically beats pure semantic by 5-15% on recall.

Semantic search is the retrieval layer in any RAG system. The quality of your semantic search directly determines whether the LLM gets the right context. If retrieval fails, it doesn't matter how good your language model is. It will generate an answer based on the wrong information, or miss the answer entirely.

The common failure modes of semantic search: embeddings trained on general text missing domain-specific meaning (legal, medical, code), queries that are too short to embed meaningfully ("password?" isn't much signal), and chunks that are too small to capture the concept being searched for. Most teams that struggle with retrieval quality are hitting one of these three, not actually an embedding model limitation. Fix your chunking and add domain-tuned embeddings before blaming the search layer.

Re-ranking is the usual quality boost on top of pure vector search. Retrieve top-k=20 by vector similarity, then re-score those 20 with a cross-encoder like Cohere Rerank or BGE-reranker-v2 to pick the best 5. Cross-encoders see the query and document together, which catches relevance signals that pure dual-encoder embeddings miss.

A B2B SaaS company runs an internal help desk for their 4,000 employees. The HR knowledge base has 1,800 articles, and employees historically found the right article 62% of the time using keyword search. Common misses: someone searches "work from another country" and misses the article titled "International Remote Work Policy."

The IT team migrates to hybrid search. Articles are chunked at 512 tokens with 50 overlap, embedded with Cohere embed-v3-english, and indexed in Qdrant. A parallel Elasticsearch BM25 index preserves keyword matching for specific policy numbers and benefit names. Query time: embed the user question, run vector search (top-20) and BM25 (top-20), fuse via RRF, then re-rank the combined top-20 with Cohere Rerank v3 to produce the final top-5.

On the same labeled eval set of 500 historical queries, recall@5 jumps from 62% (keyword only) to 88% (hybrid + rerank). The biggest gains come on intent-driven queries like "how much time off do I get after 3 years" (finds the PTO accrual table) and "reimbursement for working from home office" (finds the home office stipend article). Median latency: 290ms. Monthly infrastructure cost adds about $1,100 for the embedding and reranking API calls across roughly 120,000 searches. The tradeoff is clear: fewer employees opening support tickets because they couldn't find the policy themselves.