Published at Sep 23, 2024
AI development for all developers, not just AI experts. Build your AI app with Tiger Cloud today.
Written by Team Timescale
Vector search is a method of finding similar items in a dataset by representing data as numerical vectors and comparing them mathematically. This approach allows for discovering relationships and similarities that might not be apparent through traditional keyword-based searches.
By converting various types of data—such as text, images, or audio—into vector form, vector search enables the comparison of items based on their semantic content rather than surface-level features. This makes it particularly powerful for recommendation systems, anomaly detection, and natural language understanding.
Vector search goes beyond simple keyword matching. It allows finding relevant results based on meaning and context rather than exact text matches. This capability makes it valuable for applications where understanding semantics is crucial. You can unlock new possibilities in your data projects by grasping these concepts.
In this article, we will cover:
1. The fundamentals and mechanics of vector search
2. Implementation challenges and their solutions
3. Tools available for building sophisticated search systems
Imagine organizing a vast library. Instead of arranging books alphabetically or by genre, you categorize them based on content and themes. Books with similar concepts are placed near each other, even if their titles or authors differ. When searching for a topic, you find not just books with matching keywords but also related works exploring similar ideas. This approach allows you to discover relevant materials you might have missed with a traditional search.
Vector search operates on a similar principle, but instead of physical books, it deals with data points in a high-dimensional space, where proximity represents semantic similarity. Proximity in this space indicates similarity in meaning, powering applications like semantic search, natural language processing, and large language models.
Vector embeddings
The process involves several key steps, each crucial for enabling efficient and accurate similarity searches across large datasets. Here's a step-by-step outline of how vector search works:
Vector embedding: The process starts by converting the dataset into vectors. Each item in the dataset is transformed into a numerical representation called a vector embedding.
Vector storage: These vector embeddings are then stored in a specialized database designed to handle high-dimensional data efficiently.
Distance metric definition: A mathematical function is chosen to measure the similarity or difference between vectors. Common choices include Euclidean distance or cosine similarity.
Query processing: When a search query comes in, it's converted into a vector using the same embedding process.
Similarity computation: Using the defined metric, the system calculates the distance between the query vector and all the stored vectors.
Result ranking: Finally, the database vectors are sorted based on their distance to the query vector, with the closest (most similar) items appearing first in the results.
Visual representation of how vector search works
Vector search translates data into numerical representations called vector embeddings. These are arrays of real numbers that capture the original data's essential features or meaning.
For example, a simple vector embedding for the word "cat" might look like [0.2, -0.5, 0.8, 0.1], where each number represents some aspect of the word's meaning or usage in a given context.
To compare these vectors, we use distance metrics. Two standard metrics are:
Euclidean distance: Euclidean distance measures the straight-line distance between two points in space. For two vectors a and b in n-dimensional space, it's calculated as:
where:
a⋅b is the dot product of vectors aa and bb.
∥x∥ is the magnitude (or length) of vector xx.
The key to effective vector search lies in creating embeddings and choosing distance metrics that preserve the semantic relationships between data points. When done correctly, the geometric relationships between vectors in the embedding space correspond to meaningful relationships in the original data.
This approach allows us to sort search results based on their "closeness" to the query vector. Items with the smallest distance (or highest similarity) to the query are considered the most relevant, effectively ranking results by their semantic similarity to the search query.
Read why retrieval-augmented generation (RAG) is much more than vector search.
Several standard embedding techniques are used in vector search, each designed for specific data types. Some examples include:
Word2vec is a foundational technique for creating vector representations of words. It uses neural networks to learn word associations from large text corpora. The resulting embeddings capture semantic relationships, allowing words with similar meanings to cluster in vector space. This makes Word2vec particularly useful for tasks like semantic similarity and analogy solving.
Word2vec comes in two flavors:
Skip-gram (predicting context words from a target word)
Continuous Bag of Words (CBOW, predicting a target word from context words). The resulting embeddings capture semantic relationships, allowing operations like "king" - "man" + "woman" ≈ "queen."
This makes Word2vec valuable for tasks ranging from document classification to machine translation.
Tokenization algorithms represent a more sophisticated approach to text embedding. These methods break down text into smaller units called tokens, which can be subwords, words, or characters. Tokenization techniques, like those used in BERT or GPT models, can capture context-dependent meanings of words. This allows them to handle nuances like polysemy (words with multiple meanings) more effectively than simpler word-based models.
There are several algorithms used for tokenization. Popular examples include:
Byte Pair Encoding (BPE): Iteratively merges frequent character pairs to form subword units.
WordPiece: Similar to BPE, but uses likelihood increases as a merging criterion.
SentencePiece: Treats the input as a sequence of Unicode characters, allowing for language-agnostic tokenization.
These algorithms enable models to understand word components and handle rare or unseen words more effectively. You can find examples of using tokenization with PostgreSQL here.
Image embedding algorithms transform visual data into vector representations. These techniques often use convolutional neural networks or, more recently, transformer architectures to process images. The resulting embeddings capture various aspects of visual content, including shapes, textures, and higher-level features.
Convolutional Neural Networks (CNNs): Extract hierarchical features from images, with lower layers capturing essential elements like edges and higher layers representing more complex structures.
Vision Transformers (ViT): Adapt the transformer architecture, initially designed for text, to images by treating an image as a sequence of patches.
CLIP (Contrastive Language-Image Pre-training): Jointly trains on image-text pairs, allowing for versatile cross-modal understanding.
Embedding models derive their power from training on vast datasets, known as corpora, of text, images, or other media. This training process allows the models to learn the contextual relationships within the data, mapping similar concepts to nearby locations in the vector space. You can train a custom embedding model on your specific dataset for specialized applications, tailoring it to your domain's unique vocabulary or features. This approach can yield highly accurate results for niche fields.
Alternatively, you can leverage pre-trained models exposed to large, diverse datasets. These general-purpose models offer a balance of broad applicability and good performance across various domains, making them a practical choice for many applications.
The decision between custom training and using pre-trained models often depends on your specific needs, available resources, and the uniqueness of your data.
Implementing a vector search system involves several key components working together. Here's a high-level overview of how such a system typically operates:
Data ingestion: Raw data is ingested into the database through ETL (Extract, Transform, Load) processes. This might involve data cleaning, normalization, and formatting to ensure consistency.
Vector transformation: The embedding algorithm (e.g., BERT for text, ResNet for images) converts data into high-dimensional vectors, typically ranging from 100 to 1,000 dimensions. These dense vectors are then stored in a vector-optimized database or index structure.
Query processing: User queries undergo the same embedding process. This might involve tokenization, stop word removal, and embedding for text queries. Image queries could involve feature extraction before embedding.
Similarity search: The system efficiently finds similar vectors using approximate nearest neighbor (ANN) algorithms like hierarchical navigable small world (HNSW) or inverted file index (IVF). These algorithms trade perfect accuracy for speed, making them suitable for large-scale applications.
Result retrieval: The system retrieves the top-k most similar items, often using a combination of vector similarity and traditional filtering criteria.
Vector search system architecture
Implementing vector search comes with challenges like efficiently handling high-dimensional data and scaling to large datasets. The main types of challenges are:
Database limitations
High-dimensionality: Vector data typically consists of hundreds or thousands of dimensions. Traditional databases struggle to efficiently store and index such high-dimensional data.
Storage demands: The sheer volume of vector data can quickly overwhelm standard storage systems, requiring specialized solutions.
Computational intensity: Operations on high-dimensional vectors are computationally expensive, straining database resources.
Query language incompatibility
SQL limitations: Standard SQL lacks native support for vector operations, making it challenging to express vector similarity queries.
Integration hurdles: Combining vector search with traditional structured data queries can be complex, often requiring custom solutions or extensions.
Scalability issues
Nearest neighbor search complexity: As datasets grow, the time complexity of the exact nearest neighbor search increases dramatically, potentially becoming prohibitively slow for large-scale applications.
Curse of dimensionality: In high-dimensional spaces, the concept of "nearest" becomes less meaningful as the distance between any two points becomes more uniform.
Computation challenges
Distance calculations: Computing distances between high-dimensional vectors is inherently resource-intensive, especially when dealing with millions or billions of vectors.
Real-time constraints: Many applications require near-instantaneous results, which pressures systems to optimize for speed without sacrificing accuracy.
These barriers highlight the need for specialized solutions and optimization techniques in vector search implementations, particularly for large-scale or real-time applications.
To address the challenges of vector search, several innovative solutions have been developed:
Vector databases are specialized systems optimized for high-dimensional data. They feature efficient storage structures for dense vectors, built-in similarity search algorithms, and improved query performance for vector operations.
These databases, such as Milvus, and Weaviate, enable faster and more scalable vector search applications than traditional databases. By addressing the unique challenges of vector data, you can build sophisticated search and recommendation systems with less complexity and better performance.
Hierarchical navigable small world (HNSW) networks
Hierarchical navigable small world (HNSW) networks organize vectors in a multi-layered graph structure. Each layer contains a subset of vectors, with density increasing in lower layers. Searches start at the sparse top layer and refine through denser layers, significantly reducing the search space and improving efficiency.
Simplified HNSW structure
Centroid indexing
Centroid indexing clusters vectors and computes a centroid for each cluster. Vectors are indexed by their cluster centroids. The search process finds the nearest centroid to the query vector and then searches within the corresponding cluster. This technique effectively reduces the overall search space, particularly for large datasets.
Product quantization (PQ)
Product quantization (PQ) compresses vectors by dividing them into subvectors and quantizing each subvector independently. This approach allows for efficient storage and enables fast approximate distance computations, making it valuable for handling large-scale vector databases.
Locality-sensitive hashing (LSH)
Locality-sensitive hashing (LSH) uses specialized hash functions that map similar items to the same buckets with high probability. By reducing the search space to only the relevant buckets, LSH accelerates similarity searches in high-dimensional spaces.
These techniques significantly improve the scalability and efficiency of vector search systems. However, the field is rapidly evolving, with researchers and engineers continually developing new optimization methods to handle ever-larger datasets and more complex search requirements.
The choice of technique often depends on specific use cases, balancing factors like dataset size, query speed requirements, and accuracy needs.
The vector search ecosystem has grown significantly, offering a variety of tools to implement and optimize vector search systems. These tools generally fall into two main categories:
Vector embedding algorithms and APIs: These tools convert raw data into vector representations.
Vector databases: Specialized systems for efficiently storing, indexing, and querying vector data.
Each category serves a crucial role in the vector search pipeline. Embedding tools transform diverse data types into uniform vector formats, while vector databases manage these vectors for fast and accurate similarity searches.
The choice of tools depends on factors like data type, scale of operations, performance requirements, and integration needs with existing systems. These tools provide flexibility and community support, allowing you to customize and extend functionality as needed for your specific use cases.
Vector embedding algorithms and APIs are essential tools that transform raw data into vector representations suitable for similarity search. These solutions range from open-source libraries to cloud-based APIs, offering flexibility for various use cases. They can be used as-is or customized to meet specific organizational needs.
The OpenAI tokenization embedding API provides state-of-the-art text embeddings through a simple REST interface. It's known for its high-quality representations and ease of use, making it popular for many natural language processing tasks.
For those preferring open-source solutions, TensorFlow offers comprehensive Word2vec tutorials and implementations. These resources allow developers to understand and implement word embeddings from scratch, providing complete control over the embedding process.
Vector databases are specialized data storage and query systems that efficiently handle vector embeddings and similarity searches. They address the unique challenges of high-dimensional vector data and offer optimized performance for vector operations.
Key features of a robust vector database include support for standard similarity metrics like Euclidean distance and cosine similarity. These databases excel at managing high-dimensional data, employing advanced indexing techniques to maintain query speed as data volume grows.
An essential capability of vector databases is their integration with structured data systems. This feature enables mixed-use cases, where traditional relational queries can be combined with vector similarity searches, broadening the scope of possible applications.
While several specialized vector databases are available, solutions like Timescale aim to extend existing database systems with vector search capabilities. Timescale leverages PostgreSQL's robust foundation as a starting point and builds upon the foundational functionality of pgvector, a PostgreSQL extension for vector operations.
Timescale Cloud, which provides developers with managed PostgreSQL database services, offers an open-source PostgreSQL stack (pgvector, pgvectorscale, and pgai) for AI applications. Using this stack, developers can quickly and efficiently build AI applications on PostgreSQL.
Through these extensions, and because of its PostgreSQL foundation, Timescale Cloud offers support for vector embeddings while maintaining the flexibility and familiarity of a relational database.
At the core of Timescale's vector search implementation is an index inspired by Microsoft's DiskANN algorithm, called StreamingDiskANN. This approach enables efficient searching of large-scale vector datasets, offering a balance between search speed and accuracy. StreamingDiskANN is particularly effective for the SSD-based storage of vectors (which is cheaper than RAM) and aligns well with database infrastructure.
Timescale's vector database solution offers several key advantages:
Production-ready vector search: Timescale provides vector search capabilities designed to handle the demands of production environments. This includes high-performance indexing, efficient query processing, and the ability to handle large-scale datasets.
PostgreSQL integration: By building on PostgreSQL, Timescale leverages its reliability, extensive feature set, and broad ecosystem support. This integration allows you to use familiar SQL queries and tools for vector data. It also means you can easily extend existing PostgreSQL-based applications to incorporate vector search capabilities without needing a separate, specialized database system.
Time-based optimization: Timescale's expertise in time-series data is applied to vector search, enabling efficient time-based indexing. This feature particularly benefits applications with high, consistent data ingestion rates. It allows for optimized queries that combine vector similarity with time-based filters, crucial for real-time analytics, trend detection, and historical data analysis.
Versatility: Timescale's solution allows for hybrid queries that combine vector similarity search with traditional SQL operations, enabling complex data retrieval scenarios. This versatility is particularly valuable in applications correlating vector embeddings with structured data, such as combining semantic search results with user metadata or product information.
Performance: Timescale's vector search outperforms specialized vector databases in specific scenarios, offering faster query speeds and higher accuracy. In a benchmark test (using a dataset of 50 million Cohere embeddings of 768 dimensions each), Timescale compared the performance of PostgreSQL with pgvector and pgvectorscale against Pinecone. The results showed that compared to Pinecone’s storage-optimized index (s1), PostgreSQL with pgvector and pgvectorscale achieves 28x lower p95 latency and 16x higher query throughput for approximate nearest neighbor queries at 99 % recall.
By extending PostgreSQL into a capable vector database, Timescale aims to provide a unified solution for various data types—relational, time-series, and now vector data—all within a familiar PostgreSQL environment. This approach simplifies the data infrastructure stack for many applications, particularly those leveraging AI and machine learning technologies.
Integrating vector search into your organization's data infrastructure enables advanced AI applications such as semantic search and recommendation systems. To support these applications, vector databases have become crucial for many data-driven operations.
Effective vector search implementation hinges on a database that can efficiently handle high-dimensional data while providing robust search capabilities. Timescale combines these requirements with the familiar PostgreSQL environment.
Key features of this approach include:
High-performance vector operations optimized for large-scale datasets, as demonstrated by the benchmarks we reviewed
Seamless integration with existing PostgreSQL-based systems, leveraging its established ecosystem
The ability to perform complex queries that combine vector similarity with traditional SQL operations, addressing the hybrid search needs we identified
Efficient time-based indexing, which is particularly valuable for applications with high data ingestion rates
When selecting a vector database solution, it's important to consider the factors discussed throughout this article: data volume, query patterns, integration requirements, and existing infrastructure. These considerations will help ensure the chosen solution aligns with your organization's needs.
Want to test Timescale’s vector search potential to evaluate how it might benefit your specific use case? Pgai and pgvectorscale are open source under the PostgreSQL License and available for use in AI projects. Installation instructions are provided on the pgai GitHub repository and the pgvectorscale GitHub repository. You can also access these extensions on any database service on Timescale’s Cloud PostgreSQL platform.