Supported Models

The following models are currently supported for LLM inference deployment in Dynamiq:

Model	Parameters	Size	Architecture	Minimum AWS instance
meta-llama/Meta-Llama-3.1-8B	8.03B	16.07 GB	LlamaForCausalLM	g5.2xlarge
meta-llama/Meta-Llama-3.1-8B-Instruct	8.03B	16.07 GB	LlamaForCausalLM	g5.2xlarge
meta-llama/Meta-Llama-3.1-70B	70.6B	141.1 GB	LlamaForCausalLM	g5.48xlarge
meta-llama/Meta-Llama-3.1-70B-Instruct	70.6B	141.1 GB	LlamaForCausalLM	g5.48xlarge
meta-llama/Llama-3.2-1B	1.24B	2.47 GB	LlamaForCausalLM	g5.xlarge
meta-llama/Llama-3.2-1B-Instruct	1.24B	2.47 GB	LlamaForCausalLM	g5.xlarge
meta-llama/Llama-3.2-3B	3.21B	6.43 GB	LlamaForCausalLM	g5.xlarge
meta-llama/Llama-3.2-3B-Instruct	3.21B	6.43 GB	LlamaForCausalLM	g5.xlarge
meta-llama/Llama-2-7b-hf	6.74B	13.49 GB	LlamaForCausalLM	g5.2xlarge
meta-llama/Llama-2-7b-chat-hf	6.74B	13.49 GB	LlamaForCausalLM	g5.2xlarge
meta-llama/Llama-2-13b-hf	13B	26.03 GB	LlamaForCausalLM	g5.12xlarge
meta-llama/Llama-2-13b-chat-hf	13B	26.03 GB	LlamaForCausalLM	g5.12xlarge
meta-llama/Llama-2-70b-hf	69B	137.96 GB	LlamaForCausalLM	g5.48xlarge
meta-llama/Llama-2-70b-chat-hf	69B	137.96 GB	LlamaForCausalLM	g5.48xlarge
meta-llama/CodeLlama-7b-hf	6.74B	13.48 GB	LlamaForCausalLM	g5.2xlarge
meta-llama/CodeLlama-7b-Instruct-hf	6.74B	13.48 GB	LlamaForCausalLM	g5.2xlarge
meta-llama/CodeLlama-13b-Instruct-hf	13B	26.03 GB	LlamaForCausalLM	g5.12xlarge
meta-llama/CodeLlama-70b-Instruct-hf	69B	137.96 GB	LlamaForCausalLM	g5.48xlarge
mistralai/Mistral-Nemo-Base-2407	12.2B	24.51 GB	MistralForCausalLM	g5.12xlarge
mistralai/Mistral-Nemo-Instruct-2407	12.2B	24.51 GB	MistralForCausalLM	g5.12xlarge
mistralai/Mistral-7B-v0.1	7.24B	14.48 GB	MistralForCausalLM	g5.2xlarge
mistralai/Mistral-7B-v0.3	7.25B	14.5 GB	MistralForCausalLM	g5.2xlarge
mistralai/Mistral-7B-Instruct-v0.1	7.24B	14.48 GB	MistralForCausalLM	g5.2xlarge
mistralai/Mistral-7B-Instruct-v0.2	7.24B	14.48 GB	MistralForCausalLM	g5.2xlarge
mistralai/Mistral-7B-Instruct-v0.3	7.25B	14.5 GB	MistralForCausalLM	g5.2xlarge
HuggingFaceH4/zephyr-7b-beta	7.24B	14.48 GB	MistralForCausalLM	g5.2xlarge
microsoft/phi-3-mini-4k-instruct	3.82B	7.64GB	Phi3ForCausalLM	g5.2xlarge
microsoft/Phi-3.5-mini-instruct	3.82B	7.64GB	Phi3ForCausalLM	g5.2xlarge
google/gemma-2b	2.51B	5.01 GB	GemmaForCausalLM	g5.xlarge
google/gemma-1.1-2b-it	2.51B	5.01 GB	GemmaForCausalLM	g5.xlarge
google/gemma-7b	8.54B	17.08 GB	GemmaForCausalLM	g5.2xlarge
google/gemma-1.1-7b-it	8.54B	17.08 GB	GemmaForCausalLM	g5.2xlarge
google/gemma-2-2b	2.61B	5.23 GB	Gemma2ForCausalLM	g5.xlarge
google/gemma-2-2b-it	2.61B	5.23 GB	Gemma2ForCausalLM	g5.xlarge
google/gemma-2-9b	9.24B	18.48 GB	Gemma2ForCausalLM	g5.2xlarge
google/gemma-2-9b-it	9.24B	18.48 GB	Gemma2ForCausalLM	g5.2xlarge
google/gemma-2-27b	27.2B	54.45 GB	Gemma2ForCausalLM	g5.12xlarge
google/gemma-2-27b-it	27.2B	54.45 GB	Gemma2ForCausalLM	g5.12xlarge
Qwen/Qwen2.5-1.5B	1.54B	3.09 GB	Qwen2ForCausalLM	g5.xlarge
Qwen/Qwen2.5-1.5B-Instruct	1.54B	3.09 GB	Qwen2ForCausalLM	g5.xlarge
Qwen/Qwen2.5-3B	3.09B	6.17 GB	Qwen2ForCausalLM	g5.xlarge
Qwen/Qwen2.5-3B-Instruct	3.09B	6.17 GB	Qwen2ForCausalLM	g5.xlarge
Qwen/Qwen2.5-7B	7.62B	15.23 GB	Qwen2ForCausalLM	g5.2xlarge
Qwen/Qwen2.5-7B-Instruct	7.62B	15.23 GB	Qwen2ForCausalLM	g5.2xlarge
Qwen/Qwen2.5-14B	14.8B	29.57 GB	Qwen2ForCausalLM	g5.12xlarge
Qwen/Qwen2.5-14B-Instruct	14.8B	29.57 GB	Qwen2ForCausalLM	g5.12xlarge
Qwen/Qwen2.5-32B	32.8B	65.52 GB	Qwen2ForCausalLM	g5.12xlarge
Qwen/Qwen2.5-32B-Instruct	32.8B	65.52 GB	Qwen2ForCausalLM	g5.12xlarge
Qwen/Qwen2.5-72B	72.7B	145.41 GB	Qwen2ForCausalLM	g5.48xlarge
Qwen/Qwen2.5-72B-Instruct	72.7B	145.41 GB	Qwen2ForCausalLM	g5.48xlarge
Qwen/Qwen2-7B	7.62B	15.23 GB	Qwen2ForCausalLM	g5.2xlarge
Qwen/Qwen2-7B-Instruct	7.62B	15.23 GB	Qwen2ForCausalLM	g5.2xlarge
Qwen/Qwen2-72B	72.7B	145.41 GB	Qwen2ForCausalLM	g5.48xlarge
Qwen/Qwen2-72B-Instruct	72.7B	145.41 GB	Qwen2ForCausalLM	g5.48xlarge

VRAM Requirements

A good rule of thumb for VRAM requirements for FP16 (or half-precision models) is to have at least 2x the model size. For example, a model with 8B parameters would require at least 16GB of VRAM. However, the actual VRAM requirements may vary depending on the model and the specific task.

A better and more accurate way to determine the VRAM requirements is to use the following formula:

M = (P x (Q/8)) x 1.2

Where:

• M is the required GPU memory in GB

• P is the number of parameters in billions (e.g., 8 for meta-llama/Meta-Llama-3.1-8B or 70 for meta-llama/Meta-Llama-3.1-70B)

• Q is the precision in bits (e.g., 16 for FP16 or half precision, 8 for 8-bit quantization, or 4 for 4-bit quantization, and so on)

• 1.2 is a safety factor to account for additional memory requirements (cache overhead, etc.)

Therefore, for a model like meta-llama/Meta-Llama-3.1-8B with 8B parameters and FP16 precision, the minimum VRAM requirement would be:

M = (8 x (16/8)) x 1.2 = 16 x 1.2 = 19.2 GB