Definition

What is Vector Search?

Vector search leverages Machine Learning (ML) Vector search leverages Machine Learning (ML) to capture the meaning and context of unstructured data (such as text and images) and represent it in a digital form. It is commonly used in semantic search, where the Approximate Nearest Neighbor (ANN) algorithm is employed to find similar data. Compared to traditional keyword-based search, vector search delivers more relevant results and operates more efficiently.

Why IT Leaders Should Care About Vector Search

Why Is Vector Search Important?

How many times have you tried to find something without knowing its exact name? You may know its purpose or how to describe it, but without keywords, you’re essentially searching in the dark.

Vector search overcomes this limitation by enabling intent-based search. It provides fast and context-aware results, as vector embeddings capture synonyms and associations that represent the underlying meaning of the query. You can combine vector search with filtering and aggregation to enhance relevance through hybrid search, and integrate it with traditional scoring methods for a more powerful search experience.

How Vector Search Engines Work？

A vector search engine, also known as a vector database or semantic search engine, is designed to find the nearest neighbors of a given (vectorized) query.

Unlike traditional search methods that rely on keyword frequency, lexical similarity, or word occurrence, vector search engines use distance metrics in an embedding space to represent similarity. This makes finding relevant data equivalent to searching for the nearest neighbors of your query.

Vector Embeddings
Vector embeddings are digital representations of data and their contextual meaning, stored in high-dimensional (dense) vectors. These embeddings are typically generated by models trained on large-scale datasets, providing more accurate and relevant results. In some cases, numerical data you collect or design to represent key features of a document can also be used as embeddings. The key is to ensure efficient search capabilities.。
Learn more about text embeddings
Similarity Score
The core principle of vector search is that similar data will have similar vector representations. Once both the query and the documents are indexed using vector embeddings, you can retrieve the most similar documents to your query.
Learn more about text similarity
Artificial Neural Network (ANN) Algorithm
Traditional nearest neighbor algorithms, such as k-Nearest Neighbor (kNN), can be computationally expensive and slow, especially in high-dimensional spaces. ANN algorithms sacrifice a small amount of precision for scalability and efficiency, making them ideal for large-scale vector search applications.
Learn more about ANN

Vector Search Use Cases

Vector search is not only driving the next generation of search experiences but also unlocking new possibilities across various domains.

Leverage Machine Learning to enhance search capabilities

Semantic Search

Vector search supports semantic search or similarity search. By capturing the meaning and context of data within embeddings, it can understand user intent without requiring exact keyword matches. It can handle text documents, images, and audio. You can quickly find products that are similar or relevant to your query.

Watch the video to see how vector search enables semantic search

Recommendation
Models used to generate embeddings can continuously learn to identify similar documents and their vector representations in the embedding space. For example, an application might recommend movies or products that others who purchased the same item also liked. However, it’s important to ensure that these embeddings are derived based on certain metrics such as popularity and reputation.
Vector similarity can be combined with other metrics to achieve multiple recommendation goals. For example, you can rank product recommendations based on satisfaction scores and revenue potential.
Question Answering
When converting documents into text embeddings, it can be combined with modern Natural Language Processing (NLP) to provide full-text answers to questions. This allows users to avoid reading long manuals, and your team can provide answers more quickly.
“ question-answering” transformer model can use the text embeddings of the document knowledge base and your current question to provide the closest match as an “answer.”
How to apply QA in Elastic

Vector Search Empowers More

Don’t stop at semantic search!

Browse Unstructured Data
Search any unstructured data. You can create embeddings for text, images, audio, or sensor measurements.
How to implement image similarity search
Metadata-based Filtering
Use metadata to filter vector search results. By applying filtering conditions consistent with Approximate Nearest Neighbor (ANN) search, you can maintain recall without sacrificing speed.
Learn how to filter queries
Re-ranking Search Results
Vector similarity can be interpreted as a similarity score, which you can re-rank by combining it with other data. This includes static fields already present in the vector search database, as well as new attributes obtained through Machine Learning models.
Learn how to re-rank in Elastic
Hybrid Scoring
To further optimize, you can combine vector similarity with BM25F scoring, known as hybrid scoring. With hybrid scoring, you can rank images based on vector similarity while still achieving BM25F ranking, thus improving text relevance.
Use hybrid scoring in Elastic

Get Started

Easily apply vector search and Natural Language Processing (NLP) with Elastic

Implementing vector search and applying NLP models is not difficult. With the Elasticsearch Relevance Engine (ESRE), you get a toolkit to build AI-powered search applications that can work with Generative AI and Large Language Models (LLMs).

Using ESRE, you can build creative search applications, generate embeddings, store and search vectors, and perform semantic search through Elastic’s Learned Sparse Encoder. Learn more about using Elasticsearch as your vector database.

Semantic Search Out of the Box
Elastic’s Learned Sparse Encoder provides highly relevant out-of-the-box semantic search without domain adaptation. It can be used with a single click when configuring your search application. The model can expand queries with relevant keywords and relevance scores, learned during training, so you don’t need to configure synonyms. Unlike dense vector embeddings, they are also easy to interpret.
Read the blog post for more details
Large Language Models
Use your private data (not just publicly available training data) to provide business-specific information to Large Language Models (LLMs). Use Elasticsearch and access Generative AI through APIs and plugins integrated with your chosen LLM.
Learn how OpenAI and private data can work together
Implement Transformer Models
You can bring your own transformer models into Elastic, or use pre-trained models without the need for specialized knowledge to train them. Elastic supports HuggingFace Model Hub and various supported architectures, such as BERT, BART, ELECTRA, etc.
Read the BYOM blog post
Text Embeddings and More
Learn how to assign sentiment and other categories to your data using Elasticsearch. Apply Named Entity Recognition (NER) to improve the search experience with additional metadata.
The Application of Modern NLP in the Elastic Stack