Document Embedders
Last updated
Last updated
Document vectorization is a crucial step in the indexing workflow, transforming text data into numerical vector representations. These vectors enable efficient similarity searches, allowing the RAG application to match user queries with relevant documents based on vector proximity. This process enhances the retrieval accuracy and speed, making it a vital component of the RAG system.
Several document embedders are available, each offering unique capabilities for vectorizing text data. These embedders convert text into high-dimensional vectors, capturing semantic meanings and relationships.
Model Selection: Choose from various models, such as text-embedding-3-small, to suit your specific needs.
Dimensions: Specify the dimensionality of the vectors, which affects the granularity and detail of the representation.
Enable Caching: Option to cache embeddings for faster retrieval and reduced computational load.
Provide the split documents from the previous chunking step. The vectorizer will process these documents to generate vector embeddings.
Select the appropriate embedder and model based on your requirements. Configure the dimensions to balance between detail and computational efficiency.
The vectorizer outputs the vectorized documents, ready for storage and retrieval. These vectors are used to perform similarity searches during the inference phase.
Efficient Retrieval: Vectors enable quick similarity searches, improving the speed of information retrieval.
Enhanced Accuracy: Captures semantic relationships, ensuring that retrieved documents are contextually relevant.
Scalability: Handles large datasets efficiently, making it suitable for extensive knowledge bases.
By effectively utilizing document embedders, you can optimize your data for retrieval, ensuring that your RAG application delivers precise and contextually relevant information.
In the next section, we will explore the storage process, detailing how to save vectorized data for efficient retrieval during the inference phase.
