Vector Search Internals How Pinecone and Weaviate Actually Work Exact search is perfect but unusable. Approximate search is 1,000x faster — and finds 99.9% of the same results.

I. The Problem

You have a query vector — 768 numbers representing the meaning of a sentence. You need to find the most similar vectors in a database of 10 million documents. How?

The obvious approach: compare the query against every single vector, compute the distance for each, sort, return the closest. This is exact search. It's correct. It's also O(n) — linear in the number of vectors.

For 10 million vectors at 768 dimensions, each comparison takes ~2 microseconds. That's ~20 seconds per query. In a production system serving 1,000 queries per second, you'd need 20,000 seconds of compute per second of wall time. Physically impossible.

THE MATH 10M vectors × 768 dimensions = ~2µs per comparison × 10M = ~20 seconds per query

The problem isn't the algorithm. It's the scale. Exact search works fine for 1,000 vectors. At 10 million, it collapses.

II. The Big Idea

Here's the insight that changes everything: you don't need the exact nearest neighbor. You need a really good guess.

If the true nearest neighbor is document #4,872,341 and your search returns document #4,872,342 — which is 0.001% less similar — does the user notice? No. They're looking for relevant documents, not mathematical perfection.

This is Approximate Nearest Neighbor (ANN) search. The best ANN algorithms find 99.9% of the same results as exact search — at up to 1,000x the speed. You trade 0.1% accuracy for 1,000x speed. For any real-world application, that's not a compromise. It's the only option.

METRIC EXACT ANN Accuracy 100% 99.9% Speed (10M vectors) ~20 sec ~10 ms Production-ready? No Yes

III. HNSW: The Layered Graph

The most popular ANN algorithm. Used by Weaviate, Qdrant, Milvus, and pgvector. HNSW (Hierarchical Navigable Small World) is a layered graph structure inspired by the probability skip list, a data structure from 1990.

The analogy: imagine finding a person in a 50-story building. You don't check every room. You start at the top floor — few people, long sightlines. Find the general direction. Drop to the next floor — more people, finer detail. Keep dropping until the ground floor, where every person exists and you find your target.

LAYER 3 (top): few nodes, long links   •————————————•   Entry point. Greedy search across long edges. LAYER 1: more nodes, medium links   •——•————•————•——• LAYER 0 (bottom): all nodes, shortest links   ••••••••••••••••••••   Every vector is here. Search ends — return nearest neighbors.

At each layer, the algorithm uses greedy routing: look at connected neighbors, move to the one closest to the query, repeat until no neighbor is closer (local minimum). Then drop to the next layer, where more connections exist, and continue.

Key parameters: M (neighbors per vertex — typical: 16–48, higher = better recall, more memory), ef_construction (build quality — typical: 200), and ef_search (query-time search width — higher = better recall, slower query, tunable at runtime).

The hierarchical structure means search time is O(log n). For 10 million vectors, that's ~23 steps instead of 10 million comparisons.

IV. IVF: The Clustering Approach

A completely different strategy. Instead of building a graph, IVF (Inverted File Index) partitions the vector space into regions using k-means clustering.

Imagine a map of the United States. Instead of searching every house, you first find which state the query is in, then only search houses in that state. IVF divides all vectors into k clusters, each with a centroid. During search, find the nearest centroids, then only search vectors within those clusters.

The critical parameter is nprobe — how many clusters to search. With nprobe=1, you search one cluster (fast, might miss). With nprobe=k, you search everything (exact, slow). Most teams tune nprobe experimentally, starting around nlist/10 and increasing until they hit their target recall.

IVF is simpler than HNSW, faster to build, and uses less index memory. But for the same recall, HNSW typically delivers lower latency. Most production systems default to HNSW unless memory is the primary constraint.

V. Product Quantization: 97% Compression

Both HNSW and IVF solve the speed problem. Neither solves the memory problem. 10 million vectors at 768 dimensions, stored as 32-bit floats, require ~30GB of RAM. For a billion vectors? 3TB.

Product Quantization (PQ) compresses vectors by up to 97%:

1. Split each vector into m equal chunks (subvectors) 2. For each chunk position, run k-means clustering 3. Replace each subvector with its nearest centroid ID → 768 floats (24,576 bits) → 8 IDs (64 bits) = 384x reduction

10 million vectors: 30GB → ~80MB. That's the difference between "needs a dedicated server" and "fits in cache."

The trade-off: the compressed vector is an approximation. But PQ doesn't need to be perfect — it needs to preserve the ordering of distances. If vector A is closer than vector B originally, it should still be closer after compression. For most use cases, it is.

VI. The Combined Stack

In production, you don't pick one technique. You stack them. The most common combination is IVF + PQ — cluster first, then compress vectors within each cluster:

Step 1 — IVF: Partition vectors into clusters Step 2 — PQ: Compress vectors in each cluster Step 3 — Query: Find nearest centroids, scan nprobe clusters → Rerank: Fetch full vectors for top candidates, recompute exact distances

Pinecone's own benchmarks show IVF+PQ delivers a 92x total speed increase compared to non-quantized indexes. The rerank step is crucial: PQ gets you close, then exact distance on the top 50 candidates pushes recall from ~95% to 99.9% — adding only ~1ms.

Three layers of approximation, each adding speed without meaningfully sacrificing quality.

VII. The Aha

“You trade 0.1% accuracy for 1,000x speed. Your embeddings are already approximate — searching them perfectly is overkill.”

Every vector database — Pinecone, Weaviate, Qdrant, Milvus, pgvector — implements some combination of HNSW, IVF, and PQ under the hood. Understanding these three techniques is understanding how vector search actually works.

Appendix: Real Databases

Database Index Type Notes Pinecone Proprietary Managed serverless. Auto-selects index based on data. Weaviate HNSW Open-source. Configurable M and ef parameters. Qdrant HNSW Rust-based. HNSW with built-in scalar + PQ quantization. Milvus HNSW, IVF, PQ Most flexible. Supports all techniques and combinations. pgvector HNSW, IVFFlat PostgreSQL extension. HNSW added in v0.5.0.