Llama-3.1-8B-Instruct
Support summary
Llama-3.1-8B-Instruct is a checkpoint published by Meta, built on the LlamaForCausalLM architecture. The newest indexed releases vLLM 0.28.0 and SGLang 0.5.19 both document support. TensorRT-LLM lists its architecture, but not this exact checkpoint.
vLLM documents support in 20 of 20 indexed releases, from 0.17.0 to 0.28.0. SGLang documents support in 13 of 13 indexed releases, from 0.5.10 to 0.5.19.
- Documented hardware
- CPU, NVIDIA / CUDA, AMD / ROCm, and Intel GPU / XPU
- Documented quantization
- BF16
- Documented features
- Speculative decoding and Tool calling
- License
- llama3.1
Hardware, quantization and features count only statements about this checkpoint; engine-wide capabilities are listed with each release.
Hugging Face model card
- Publisher
- Meta
- Task
- text generation
- License
- llama3.1
- Library
- transformers
- Architecture
- LlamaForCausalLM
- Weight tensor elements
- 8B
- Tensor types
- BF16
- Languages
- en, de, fr, it, pt, hi, es, th
- Base models
- meta-llama/Meta-Llama-3.1-8B
- Weight access
- Publisher approval or terms required
- Repository created
- 2024-07-18
Model notes and sources
Tool calling has an exact vLLM example. Keep its parser and chat template together.
- vLLM v0.28.0 · docs/features/tool_calling.md · 2 indexed revisions, v0.27.1 to v0.28.0
- meta-llama/Llama-3.1-8B-Instruct publisher metadata
- vLLM v0.28.0 · docs/models/hardware_supported_models/cpu.md · 20 indexed revisions, v0.17.0 to v0.28.0
- vLLM v0.28.0 · docs/models/hardware_supported_models/xpu.md · 20 indexed revisions, v0.17.0 to v0.28.0
- SGLang v0.5.19 · docs/docs/hardware-platforms/xpu.mdx · 3 indexed revisions, v0.5.17 to v0.5.19
- SGLang v0.5.16 · docs_new/docs/hardware-platforms/xpu.mdx · 8 indexed revisions, v0.5.11 to v0.5.16
- SGLang v0.5.10.post1 · docs/platforms/xpu.md · 2 indexed revisions, v0.5.10 to v0.5.10.post1
- meta-llama/Llama-3.1-8B-Instruct · publisher model card
- meta-llama/Llama-3.1-8B-Instruct · Hugging Face metadata
- vllm · models/meta-llama/Llama-3.1-8B-Instruct.yaml
- sglang · docs/src/snippets/specbundle/specbundle-deployment.jsx
- Reviewed
Engine support
3 engines
| Engine | Release | Checkpoint support | Evidence scope | Last known supported |
|---|---|---|---|---|
| vLLM | Version not pinnedRecipe / cookbook | Documented recipeSource: vllm · models/meta-llama/Llama-3.1-8B-Instruct.yamlReviewed | View 0.28.0 | |
Features, dependencies and evidence for vLLM 0.28.0 | ||||
| SGLang | 0.5.19Newest indexed | Checkpoint evidenceSource: SGLang v0.5.19 · docs/docs/hardware-platforms/xpu.mdxReviewed | 0.5.19This release | |
Features, dependencies and evidence for SGLang 0.5.19 | ||||
| TensorRT-LLM | 1.2.1Newest indexed | Architecture listed; exact checkpoint unknown | No supported release indexed | |
Features, dependencies and evidence for TensorRT-LLM 1.2.1 | ||||