Two years of vector search at Notion: 10x scale, 1/10th cost

Summary

When we launched Notion AI Q&A in November 2023, we knew we were embarking on an ambitious journey. What we didn't fully anticipate was just how quickly we'd need to scale, or how dramatically we could optimize our costs once we hit our stride. This is the story of how we scaled our vector search infrastructure by 10x while simultaneously reducing costs by 90 percent over the past two years.What is vector search and why does Notion AI use it?Traditional keyword search matches exact words, so a query like “team meeting notes” can miss content titled “group standup summary”, even though they mean the same thing.Vector search fixes this by turning text into semantic embeddings: points in a high-dimensional space where similar ideas cluster together. This lets us retrieve relevant content based on meaning instead of being limited to exact phrasing.This is essential for Notion AI, which answers natural-language questions by searching content from across a user’s workspace and connected tools like Slack and Google Drive.Part 1: Scaling to millions of workspacesWhen we launched in November 2023, our ingestion indexing pipeline had two paths:Offline path: Batch jobs running on Apache Spark that chunk existing documents, generate embeddings via API, and bulk-load vectors into our vector databaseOnline path: Real-time updates via Kafka consumers that process individual page edits as they happenThis dual-path architecture let us onboard large workspaces efficiently while keeping live workspaces up-to-date with sub-minute latency.Our vector database ran on dedicated “pod” clusters that coupled storage and compute, and was able to handle large-scale multi-tenant workloads. We designed a sharding strategy similar to our Postgres setup, utilizing workspace ID as a partitioning key and routing to the correct index using range based partitioning. A single config held references to all the shards. The launch generated immediate and overwhelming demand. We quickly accumulated a waitli...