Supported Models

The following models are currently supported for LLM inference deployment in Dynamiq:

Model

Parameters

Size

Architecture

Minimum AWS instance

VRAM Requirements

A good rule of thumb for VRAM requirements for FP16 (or half-precision models) is to have at least 2x the model size. For example, a model with 8B parameters would require at least 16GB of VRAM. However, the actual VRAM requirements may vary depending on the model and the specific task.

A better and more accurate way to determine the VRAM requirements is to use the following formula:

M = (P x (Q/8)) x 1.2

Where:

M is the required GPU memory in GB
P is the number of parameters in billions (e.g., 8 for meta-llama/Meta-Llama-3.1-8B or 70 for meta-llama/Meta-Llama-3.1-70B)
Q is the precision in bits (e.g., 16 for FP16 or half precision, 8 for 8-bit quantization, or 4 for 4-bit quantization, and so on)
1.2 is a safety factor to account for additional memory requirements (cache overhead, etc.)

Therefore, for a model like meta-llama/Meta-Llama-3.1-8B with 8B parameters and FP16 precision, the minimum VRAM requirement would be:

M = (8 x (16/8)) x 1.2 = 16 x 1.2 = 19.2 GB

PreviousFine-tuned Adapters NextVector Databases

Last updated 24 days ago