Document Embedders

Why Document Vectorization is Important

Document vectorization is a crucial step in the indexing workflow, transforming text data into numerical vector representations. These vectors enable efficient similarity searches, allowing the RAG application to match user queries with relevant documents based on vector proximity. This process enhances the retrieval accuracy and speed, making it a vital component of the RAG system.

Document Embedders

Several document embedders are available, each offering unique capabilities for vectorizing text data. These embedders convert text into high-dimensional vectors, capturing semantic meanings and relationships.

Available Embedders

Key Features of the Document Embedder

• Model Selection: Choose from various models, such as text-embedding-3-small, to suit your specific needs.

• Dimensions: Specify the dimensionality of the vectors, which affects the granularity and detail of the representation.

• Enable Caching: Option to cache embeddings for faster retrieval and reduced computational load.

How to Use the Document Vectorizer

1. Input

Provide the split documents from the previous chunking step. The vectorizer will process these documents to generate vector embeddings.

2. Configuration

Select the appropriate embedder and model based on your requirements. Configure the dimensions to balance between detail and computational efficiency.

3. Output

The vectorizer outputs the vectorized documents, ready for storage and retrieval. These vectors are used to perform similarity searches during the inference phase.

Benefits of Document Vectorization

• Efficient Retrieval: Vectors enable quick similarity searches, improving the speed of information retrieval.

• Enhanced Accuracy: Captures semantic relationships, ensuring that retrieved documents are contextually relevant.

• Scalability: Handles large datasets efficiently, making it suitable for extensive knowledge bases.

By effectively utilizing document embedders, you can optimize your data for retrieval, ensuring that your RAG application delivers precise and contextually relevant information.

In the next section, we will explore the storage process, detailing how to save vectorized data for efficient retrieval during the inference phase.