Run Llama-3.2-1B

Model Information

The Meta Llama 3.2 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction-tuned generative models in 1B and 3B sizes (text in/text out). The Llama 3.2 instruction-tuned text only models are optimized for multilingual dialogue use cases, including agentic retrieval and summarization tasks. They outperform many of the available open source and closed chat models on common industry benchmarks.

Model Developer: Meta

Model Architecture: Llama 3.2 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.

	Training Data	Params	Input modalities	Output modalities	Context Length	GQA	Shared Embeddings	Token count	Knowledge cutoff
Llama 3.2 (text only)	A new mix of publicly available online data.	1B (1.23B)	Multilingual Text	Multilingual Text and code	128k	Yes	Yes	Up to 9T tokens	December 2023
		3B (3.21B)	Multilingual Text	Multilingual Text and code

Supported Languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai are officially supported. Llama 3.2 has been trained on a broader collection of languages than these 8 supported languages. Developers may fine-tune Llama 3.2 models for languages beyond these supported languages, provided they comply with the Llama 3.2 Community License and the Acceptable Use Policy. Developers are always expected to ensure that their deployments, including those that involve additional languages, are completed safely and responsibly.

Llama 3.2 Model Family: Token counts refer to pretraining data only. All model versions use Grouped-Query Attention (GQA) for improved inference scalability.

Model Release Date: Sept 25, 2024

Status: This is a static model trained on an offline dataset. Future versions may be released that improve model capabilities and safety.

License: Use of Llama 3.2 is governed by the Llama 3.2 Community License (a custom, commercial license agreement).

Feedback: Where to send questions or comments about the model Instructions on how to provide feedback or comments on the model can be found in the model README. For more technical information about generation parameters and recipes for how to use Llama 3.2 in applications, please go here.

Intended Use

Intended Use Cases: Llama 3.2 is intended for commercial and research use in multiple languages. Instruction tuned text only models are intended for assistant-like chat and agentic applications like knowledge retrieval and summarization, mobile AI powered writing assistants and query and prompt rewriting. Pretrained models can be adapted for a variety of additional natural language generation tasks.

Out of Scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Llama 3.2 Community License. Use in languages beyond those explicitly referenced as supported in this model card.

How to use

This repository contains two versions of Llama-3.2-1B, for use with transformers and with the original llama codebase.

Use with transformers

Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.

Make sure to update your transformers installation via pip install --upgrade transformers.

import torch
from transformers import pipeline

model_id = "meta-llama/Llama-3.2-1B"

pipe = pipeline(
    "text-generation", 
    model=model_id, 
    torch_dtype=torch.bfloat16, 
    device_map="auto"
)

pipe("The key to life is")

Use with llama

Please, follow the instructions in the repository.

To download Original checkpoints, see the example command below leveraging huggingface-cli:

huggingface-cli download meta-llama/Llama-3.2-1B --include "original/*" --local-dir Llama-3.2-1B

Hardware and Software

Training Factors: We used custom training libraries, Meta's custom built GPU cluster, and production infrastructure for pretraining. Fine-tuning, annotation, and evaluation were also performed on production infrastructure.

Training Energy Use: Training utilized a cumulative of 916k GPU hours of computation on H100-80GB (TDP of 700W) type hardware, per the table below. Training time is the total GPU time required for training each model and power consumption is the peak power capacity per GPU device used, adjusted for power usage efficiency.

Training Greenhouse Gas Emissions: Estimated total location-based greenhouse gas emissions were 240 tons CO2eq for training. Since 2020, Meta has maintained net zero greenhouse gas emissions in its global operations and matched 100% of its electricity use with renewable energy; therefore, the total market-based greenhouse gas emissions for training were 0 tons CO2eq.

	Training Time (GPU hours)	Logit Generation Time (GPU Hours)	Training Power Consumption (W)	Training Location-Based Greenhouse Gas Emissions (tons CO2eq)	Training Market-Based Greenhouse Gas Emissions (tons CO2eq)
Llama 3.2 1B	370k	-	700	107	0
Llama 3.2 3B	460k	-	700	133	0
Total	830k	86k		240	0

The methodology used to determine training energy use and greenhouse gas emissions can be found here. Since Meta is openly releasing these models, the training energy use and greenhouse gas emissions will not be incurred by others.

Training Data

Overview: Llama 3.2 was pretrained on up to 9 trillion tokens of data from publicly available sources. For the 1B and 3B Llama 3.2 models, we incorporated logits from the Llama 3.1 8B and 70B models into the pretraining stage of the model development, where outputs (logits) from these larger models were used as token-level targets. Knowledge distillation was used after pruning to recover performance. In post-training we used a similar recipe as Llama 3.1 and produced final chat models by doing several rounds of alignment on top of the pre-trained model. Each round involved Supervised Fine-Tuning (SFT), Rejection Sampling (RS), and Direct Preference Optimization (DPO).

Data Freshness: The pretraining data has a cutoff of December 2023.

Benchmarks - English Text

In this section, we report the results for Llama 3.2 models on standard automatic benchmarks. For all these evaluations, we used our internal evaluations library.

Base Pretrained Models

Category	Benchmark	# Shots	Metric	Llama 3.2 1B	Llama 3.2 3B	Llama 3.1 8B
General	MMLU	5	macro_avg/acc_char	32.2	58	66.7
	AGIEval English	3-5	average/acc_char	23.3	39.2	47.8
	ARC-Challenge	25	acc_char	32.8	69.1	79.7
Reading comprehension	SQuAD	1	em	49.2	67.7	77
	QuAC (F1)	1	f1	37.9	42.9	44.9
	DROP (F1)	3	f1	28.0	45.2	59.5
Long Context	Needle in Haystack	0	em	96.8	1	1

Instruction Tuned Models

Capability		Benchmark	# Shots	Metric	Llama 3.2 1B	Llama 3.2 3B	Llama 3.1 8B
General		MMLU	5	macro_avg/acc	49.3	63.4	69.4
Re-writing		Open-rewrite eval	0	micro_avg/rougeL	41.6	40.1	40.9
Summarization		TLDR9+ (test)	1	rougeL	16.8	19.0	17.2
Instruction following		IFEval	0	avg(prompt/instruction acc loose/strict)	59.5	77.4	80.4
Math		GSM8K (CoT)	8	em_maj1@1	44.4	77.7	84.5
		MATH (CoT)	0	final_em	30.6	47.3	51.9
Reasoning		ARC-C	0	acc	59.4	78.6	83.4
		GPQA	0	acc	27.2	32.8	32.8
		Hellaswag	0	acc	41.2	69.8	78.7
Tool Use		BFCL V2	0	acc	25.7	67.0	70.9
		Nexus	0	macro_avg/acc	13.5	34.3	38.5
Long Context		InfiniteBench/En.QA	0	longbook_qa/f1	20.3	19.8	27.3
		InfiniteBench/En.MC	0	longbook_choice/acc	38.0	63.3	72.2
		NIH/Multi-needle	0	recall	75.0	84.7	98.8
Multilingual		MGSM (CoT)	0	em	24.5	58.2	68.9

Multilingual Benchmarks

Category	Benchmark	Language	Llama 3.2 1B	Llama 3.2 3B	Llama 3.1 8B
General	MMLU (5-shot, macro_avg/acc)	Portuguese	39.82